On the Estimation of Mutation Rate Based on Coalescence Genealogy

Estimating gene mutation rates using DNA data from a population since its coalescence is an important part in the study of evolutionary genetic history, and is of its own interest also. There are a number of existing methods for this goal and are successful in application, but formal asymptotic study is absent except for the Watterson estimator, such as consistency of the estimator and its rate. Since the observed data are genetically dependent, the problem is non-standard. Here we investigate the asymptotic property for the estimation of mutation rate in a given population under three common cases in practice, estimations based on coalescent tree, rooted tree and the total number of observed segregation sites only. We show, in each case, the strong consistency of the estimate, and its asymptotic normality with a very slow rate of (log n), in contrast to the standard rate of nfor independently identically distributed data. Although this result is known several decades ago, we rediscovered this fact without knowing the previous results and the settings and consitions we used are not all the same as those previous studies. We also propose a simple simulation based method for the estimation of the mutation rate using DNA data. The result is illustrated by a segment of the mitochondrial data from an Amerindian tribal population, compared to some of the commonly used existing methods, and found to be consistent with these methods.